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DETAILED ACTION 

Drawings 

1 . Figures 1 and 2 should be designated by a legend such as --Prior Art-- because 
only that which is old is illustrated. See MPEP § 608.02(g). Corrected drawings in 
compliance with 37 CFR 1 .121(d) are required in reply to the Office action to avoid 
abandonment of the application. The replacement sheet(s) should be labeled 
"Replacement Sheet" in the page header (as per 37 CFR 1 .84(c)) so as not to obstruct 
any portion of the drawing figures. If the changes are not accepted by the examiner, the 
applicant will be notified and informed of any required corrective action in the next Office 
action. The objection to the drawings will not be held in abeyance. 

Specification 

2. The disclosure is objected to because of the following informalities: the various 
docket numbers on pages 2 and 3 of the specification should be replaced with 
application serial numbers. 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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4. This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 1 03(c) and potential 35 U.S.C. 1 02(e), (f) or (g) 
prior art under 35 U.S.C. 1 03(a). 

5. Claims 1-5, 9-11, 13, 18-19 and 23-28 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over the combination of U.S. Patent 5,546,538 to Cobbley et al. 
(hereinafter referred to as "Cobbley") and U.S. Patent Application 20040126017 to Seni 
etal. (hereinafter "Seni"). 

6. As to claim 1 , Cobbley discloses a method of providing computer-based 
recognition of natural language data, comprising the steps of: 

generating natural language data using an input device (Fig.1 , element 20; Fig. 2, 
element 20; column 3, lines 20-21 ; column 5, lines 24-30; the natural language data is 
the handwriting data); and, 

transmitting the natural language data to a server via a network (column 4, lines 
45-47; Fig. 2, element 35; column 4, lines 48-49; column 5, line 61 to column 6, line 1 ); 

wherein, the server is programmed and configured to process the natural 
language data using a recognizer residing on the server (column 6, lines 26-30) to 
produce intermediate format data (column 6, lines 30-34; the intermediate format data is 



Application/Control Number: 1 0/51 0,392 Page 4 

Art Unit: 2624 

encoded text), and is further programmed and configured to transmit the intermediate 
format data to an application (Fig. 7, element 307; column 7, lines 5-15), and 

further wherein, the intermediate format data is decoded into computer-readable 
format data (column 3, lines 30-32; Fig.1 , element 21 ; column 5, lines 38-43; Fig.4a, 
element 103, steps 2 and 3; the device 20 inherently decodes the encoded text since it 
receives encoded text and displays the text 21). 

Cobbley does not disclose decoding using context information associated with 
the application. However, this is well known in the art. For example, Seni, in the same 
environment, discloses using "grammars" which provide a context information when 
decoding text ([0023], lines 8-13; [0034]; [0058], lines 1-3; [0066]). Seni's states that 
context permits conversion of handwritten input into text strings that are pertinent or 
relevant to the circumstances or surroundings in which an expected text string is being 
used (last sentence of [0066]). Therefore, it would have been obvious to one of 
ordinary skill in the art to modify Cobbley according to Seni, in order to obtain this 
advantage. 

7. As to claim 2, Cobbley discloses a method for computer-based recognition of 
natural language data, the method implemented on a network (Fig. 3; LAND 42) and 
comprising the steps of: 

obtaining natural language data using an input device (Fig. 1 , element 20; Fig. 2, 
element 20; column 3, lines 20-21 ; column 5, lines 24-30; the natural language data is 
the handwriting data); 
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receiving the natural language data on a server via the network (column 6, lines 
26-29); 

processing the natural language data using a recognizer residing on the server to 
produce intermediate format data (column 6, lines 30-34; the intermediate format data is 
encoded text); 

transmitting the intermediate format data to an application (Fig. 7, element 307; 
column 7, lines 5-15); and, 

decoding the intermediate format data into computer-readable format data 
(column 3, lines 30-32; Fig.1 , element 21 ; column 5, lines 38-43; Fig.4a, element 103, 
steps 2 and 3; the device 20 inherently decodes the encoded text since it receives 
encoded text and displays the text 21 ). 

Cobbley does not disclose decoding using context information associated with 
the application. However, this is well known in the art. For example, Seni, in the same 
environment, discloses using "grammars" which provide a context information when 
decoding text ([0023], lines 8-13; [0034]; [0058], lines 1-3; [0066]). Seni's states that 
context permits conversion of handwritten input into text strings that are pertinent or 
relevant to the circumstances or surroundings in which an expected text string is being 
used (last sentence of [0066]). Therefore, it would have been obvious to one of 
ordinary skill in the art to modify Cobbley according to Seni, in order to obtain this 
advantage. 
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8. As to claim 3, Cobbley further discloses the method as claimed in claim 1 or 2, 
wherein the natural language data is digital ink (in Fig.1, the result of writing is digital 
ink) or speech (column 8, lines 24-32). Seni also teaches digital ink ([0023], lines 6-7). 

9. Regarding claim 4, Cobbley discloses the method as claimed in claim 1 or 2, but 
does not disclose wherein processing the natural language data includes one or more 
of: normalizing the data; segmenting the data; and classifying the data. However, this is 
well known in the art as evidenced by Seni ([0024], lines 6-1 1 ). It would have been 
obvious to one of ordinary skill in the art to modify Cobbley according to Seni to employ 
classifying of the data in order to improve conversion to text. 

10. Regarding claim 5, Cobbley discloses the method as claimed in claim 1 or 2, 
wherein the recognizer is implemented using software or hardware (column 6, lines 26- 
34; the recognizer is essentially software running on hardware). 

1 1 . With regard to claim 9, Cobbley discloses the method as claimed in claim 1 or 2, 
wherein the natural language data is derived from image processing (Fig. 1; the 
handwriting capture qualifies as image processing). 

12. As to claim 10, Cobbley discloses the method as claimed in claim 1 or 2, but 
does not disclose wherein the application is remote to both the input device and the 
server. However, Seni teaches forwarding to any other computer program or 
application ([0023], last sentence), and another computer or computer program ([0028], 
last sentence). Seni also shows another computer on the network (Fig.1), which is 
remote from the other two. It would have been obvious to one of ordinary skill in the art 
to modify Cobbley's invention to have an application that is remote to both the input 
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device and the server based on the teachings of Seni because this would make more 
effective use of distributed processing. 

1 3. Regarding claim 1 1 , Cobbley discloses the method as claimed in claim 1 or 2, 
wherein the application resides on the server (this could read on any number of things 
in Cobbley. For example, the application could be that which operates the interface of 
the computational server with the LAN (Fig.2). 

14. Regarding claim 1 3, Cobbley discloses the method as claimed in claim 1 or 2, 
wherein the recognizer can be trained for a specific user (column 6, lines 48-61). 

15. As to claim 18, Cobbley discloses a method for computer-based recognition of 
natural language data, comprising the steps of: 

receiving natural language data at a server from a remote input device (column 
6, lines 26-29; Fig.1 , element 20; Fig.2, element 20; column 3, lines 20-21 ; column 5, 
lines 24-30; the natural language data is the handwriting data); 

processing the natural language data using a recognizer residing on the server to 
produce intermediate format data (column 6, lines 30-34; the intermediate format data is 
encoded text); and, 

transmitting the intermediate format data to an application (Fig. 7, element 307; 
column 7, lines 5-15); 

wherein, the application is programmed and configured to decode the 
intermediate format data into computer-readable format data (column 3, lines 30-32; 
Fig.1, element 21; column 5, lines 38-43; Fig.4a, element 103, steps 2 and 3; the device 
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20 inherently decodes the encoded text since it receives encoded text and displays the 
text 21). 

Cobbley does not disclose decoding using context information associated with 
the application. However, this is well known in the art. For example, Seni, in the same 
environment, discloses using "grammars" which provide a context information when 
decoding text ([0023], lines 8-13; [0034]; [0058], lines 1-3; [0066]). Seni's states that 
context permits conversion of handwritten input into text strings that are pertinent or 
relevant to the circumstances or surroundings in which an expected text string is being 
used (last sentence of [0066]). Therefore, it would have been obvious to one of 
ordinary skill in the art to modify Cobbley according to Seni, in order to obtain this 
advantage. 

16. Claim 1 9 is similar to claim 18. The discussion provide above for claim 1 8 is 
therefore applicable to claim 19. 

17. Regarding claim 23, Cobbley discloses a system for computer-based recognition 
of natural language data, the system implemented on a network and comprising: 

a server to receive natural language data generated by an input device via the 
network (column 6, lines 26-29); and, 

a recognizer residing on the server to process the natural language data to 
produce intermediate format data (column 6, lines 30-34; the intermediate format data is 
encoded text); wherein, 

an application receives the intermediate format data and decodes the 
intermediate format data into computer-readable format data (column 3, lines 30-32; 
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Fig.1, element 21; column 5, lines 38-43; Fig. 4a, element 103, steps 2 and 3; the 
application running on the device 20 inherently decodes the encoded text since it 
receives encoded text and displays the text 21). 

Cobbley does not disclose decoding using context information associated with 
the application. However, this is well known in the art. For example, Seni, in the same 
environment, discloses using "grammars" which provide a context information when 
decoding text ([0023], lines 8-13; [0034]; [0058], lines 1-3; [0066]). Seni's states that 
context permits conversion of handwritten input into text strings that are pertinent or 
relevant to the circumstances or surroundings in which an expected text string is being 
used (last sentence of [0066]). Therefore, it would have been obvious to one of 
ordinary skill in the art to modify Cobbley according to Seni, in order to obtain this 
advantage. 

18. As to claim 24, Cobbley discloses a system for computer-based recognition of 
natural language data, the system implemented on a network and comprising: 

an input device to generate natural language data (Fig.1, element 20; Fig. 2, 
element 20; column 3, lines 20-21 ; column 5, lines 24-30; the natural language data is 
the handwriting data); 

a server to receive the natural language data via the network (column 6, lines 26- 

29); 

a recognizer residing on the server to process the natural language data to 
produce intermediate format data (column 6, lines 30-34; the intermediate format data is 
encoded text); and, 
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an application to receive the intermediate format data and to decode the 
intermediate format data into computer-readable format data (column 3, lines 30-32; 
Fig.1 , element 21 ; column 5, lines 38-43; Fig.4a, element 1 03, steps 2 and 3; the 
application running on the device 20 inherently decodes the encoded text since it 
receives encoded text and displays the text 21 ). 

Cobbley does not disclose decoding using context information associated with 
the application. However, this is well known in the art. For example, Seni, in the same 
environment, discloses using "grammars" which provide a context information when 
decoding text ([0023], lines 8-13; [0034]; [0058], lines 1-3; [0066]). Seni's states that 
context permits conversion of handwritten input into text strings that are pertinent or 
relevant to the circumstances or surroundings in which an expected text string is being 
used (last sentence of [0066]). Therefore, it would have been obvious to one of 
ordinary skill in the art to modify Cobbley according to Seni, in order to obtain this 
advantage. 

19. As to claim 25, Cobbley discloses the system as claimed in claim 23 or 24, 
wherein the input device is a pen-based input device (note Fig.1 , element 25). 

20. As to claim 26, Cobbley discloses the system as claimed in claim 23 or 24, 
wherein the input device includes a microphone (column 8, line 28). 

21 . As to claim 27, Cobbley discloses the system as claimed in claim 23 or 24, but 
does not disclose wherein the intermediate format data is transmitted to more than one 
application. However, Seni teaches forwarding to any other computer program or 
application ([0023], last sentence), and another computer or computer program ([0028], 
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last sentence). It would have been obvious to one of ordinary skill in the art to transmit 
the intermediate format data of Cobbley to more than one application based on the 
teachings of Seni because this would permit multiple users or devices to utilize the 
system. 

22. Regarding claim 28, Cobbley discloses the system as claimed in claim 23 or 24, 
wherein the application initiates the processing of the natural language data (this could 
be interpreted in a number of ways. For example this reads on Cobbley's input device 
accepting handwriting as described in column 5, lines 24-30. As another example, this 
could read on Cobbley's device passing the handwriting to the communication 
subsystem as described in column 5, lines 35-40). 

23. Claim 29 is rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Cobbley, Seni and U.S. Patent 5,455,872 to Bradley. 

24. Regarding claim 29, neither Cobbley nor Seni disclose that the system includes a 
recognizer manager to select a recognizer from a plurality of recognizers. However, this 
is old and well known in the art. For example, Bradley discloses a system for selecting 

a recognizer from a plurality of recognizers (column 2, lines 36-38). This improves and 
enhances recognition, as explained by Bradley (column 2, lines 18-25). Therefore, it 
would have been obvious to one of ordinary skill in the art to modify the Cobbley-Seni 
system according to Bradley. 
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25. Claims 6-8 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over the combination of Cobbley, Seni and U.S. Patent 7,336,827 to Geiger et al. 
(hereinafter referred to as "Geiger"). 

26. Regarding claim 6, Cobbley and Seni disclose the method as claimed in claim 1 
or 2, however neither discloses wherein the intermediate format data is a Directed 
Acyclic Graph (DAG) data structure. However, this is well known in the art. For 
example, Geiger discloses a Directed Acyclic Graph data structure (column 18, line 63 
to column 19, line 12). Employing the DAG data structure as taught by Geiger would 
have provided improved recognition processing. Therefore, it would have been obvious 
to one of ordinary skill in the art to modify the Cobbley-Seni system according to Geiger. 

27. Regarding claim 7, Geiger further discloses wherein the DAG data structure is a 
matrix containing the processing results of segments of natural language data (column 
19, lines 4-12). 

28. Regarding 8, Geiger does not disclose wherein the intermediate format data 
includes segmented time-series classifier data. However, this is well known in the art 
as evidenced by Geiger. Geiger teaches the capturing handwritten characters as 
coordinate information and time information (column 7, lines 3-5). The data is 
segmenting (column 12, lines 3-34). Further, the characters are classified (column 4, 
lines 61-62). It would have been obvious to one ordinary skill in the art to modify the 
Cobbley-Seni method according to Geiger because this would have permitted improved 
recognition of characters. 



Application/Control Number: 1 0/51 0,392 Page 1 3 

Art Unit: 2624 

29. As to claim 12, Seni does not disclose wherein the context information is a user 
dictionary. However, this is well known in the art. For example, Geiger teaches a user 
dictionary (column 21 , lines 48-52). Use of a dictionary would improve character 
recognition, and therefore it would have been obvious to one of ordinary skill in the art 
to modify Seni according to Geiger. 

30. Claims 14-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
the combination of Cobbley, Seni and U.S. Patent 7,289,685 to Wolff et al. (hereinafter 
referred to as "Wolff'). 

31 . As to claim 14, neither Cobbley nor Seni disclose wherein the input device is 
associated with a paper-based interface provided with coded markings. However, this 
is well known in the art. For example, Wolff teaches this (Fig. 2; column 3, lines 45-58; 
column 4, lines 30-37 and 48-54). Employing the paper-based interface taught by Wolff 
to the Cobbley-Seni method would increase versatility, in particular with Seni's use of 
forms. Therefore it would have been obvious to modify the Cobbley-Seni method 
according to Wolff. 

32. As to claim 15, Wolff does not teach wherein the coded markings are a pattern of 
infrared markings. However, the Examiner takes official notice that coded markings are 
well known in the art. It would have been obvious to one or ordinary skill in the art to 
utilize such markings because it would provide a "cleaner" and more user-friendly form. 

33. Regarding claim 16, none of Cobbley, Seni and Wolff disclose an optically 
imaging pen as an input device. However, the Examiner takes official notice that 
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optically imaging pens are well known in the art. It would have been obvious to one or 
ordinary skill in the art to utilize an optically imaging pen because it would allow 
improved portability when using paper-based forms. 

34. As to claim 17, Wolff further teaches wherein each paper-based interface is 
uniquely identified and stored on a network server (column 6, lines 63-64). It would 
have been obvious to implement this feature of Wolff in order to accommodate any 
number of forms for a variety of applications. 



Allowable Subject Matter 

35. Claims 20-22 are allowed. 

36. The following is a statement of reasons for the indication of allowable subject 
matter: Claim 20 recites "receiving in a server, indicating data. ..regarding. ..and at least 
one of a position and a movement of the sensing device relative to the paper-based 
document." The claim goes on to require "processing the indicating data. ..to produce 
intermediate format data," "transmitting the intermediate format data to an application," 
and "wherein the application decodes the intermediate format data into computer 
readable format using context information associated with the paper-based document." 
While the cited prior art teach various aspects of the claim, they do not teach the above 
mentioned features. Cobbley and Seni variously teach intermediate format data, 
context information, a server, etc., as described in previous rejections. Wolff teaches a 
paper-based document. However, none of these references teach the invention of 
claim 20. Claim 21 recites similar features, and is therefore also allowable. 
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Citation of Relevant Prior Art 

37. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

U.S. Patent 20050013487 to Clary et al. is cited for disclosing a system which 
captures handwriting on one computing device and transfers the handwriting to another 
computing device for recognition. 

U.S. Patent 6,456,740 to Carini et al. is cited for disclosing a system for 
identifying form type in a handwriting recognition based form completion system. 

U.S. Patent 6,343,148 to Nagy discloses a system which utilizes a server on a 
network to provide recognition services for handwriting input into a PDA. The invention 
is similar to that provided by Cobbley and Seni. 

U.S. Patent 5982,928 to Shimada et al. is cited for disclosing a system which 
utilizes dictionaries for performing character recognition. 

Contact Information 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JON CHANG whose telephone number is (571)272- 
7417. The examiner can normally be reached on M-F 8:00 a.m. -6:00 p.m.. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta can be reached on (571)272-7453. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Jon Chang / 

Primary Examiner, Art Unit 2624 



